Papers with ranking accuracy
Permutative Preference Alignment from Listwise Ranking of Human Judgments (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to align Large Language Models with human preferences are based on the Bradley-Terry model, but when multiple responses are available, the B-T model fails to guarantee an accurate list ranking of the responses. |
| Approach: | They propose an offline listwise approach that incorporates the Normalized Discounted Cumulative Gain (NDCG) as an alternative training objective for LLM alignment. |
| Outcome: | The proposed approach outperforms existing pairwise and listwise methods on evaluation sets and general benchmarks such as AlpacaEval. |
Neural Readability Pairwise Ranking for Sentences in Italian Administrative Language (2022.aacl-main)
Copied to clipboard
| Challenge: | Automatic Readability Assessment (ARA) aims at assigning a complexity level to a text, which could help improve the accessibility to information in specific domains, such as the administrative one. |
| Approach: | They propose to use a Neural Pairwise Ranking Model (NPRM) to assess Italian administrative texts by rewriting sentences using three different rewrite strategies. |
| Outcome: | The proposed model is effective in zero-shot scenarios (0.78) and the best results are obtained by adding in-domain data (achieving perfect performance for such sentence pairs). |
MOSPC: MOS Prediction Based on Pairwise Comparison (2023.acl-short)
Copied to clipboard
| Challenge: | et al., 2016a) show that MOS prediction model can improve ranking accuracy of speech quality. |
| Approach: | They propose a general framework for MOS prediction based on pair comparison . they use C-Mixup algorithm to enhance generalization performance of MOSPC . |
| Outcome: | The proposed model outperforms baselines on most correlation coefficient metrics . it also surpasses the strong baseline in ranking accuracy on each fine-grained segment. |
Simple and Effective Unsupervised Redundancy Elimination to Compress Dense Vectors for Passage Retrieval (2021.emnlp-main)
Copied to clipboard
| Challenge: | Dense passage retrieval improves ranking accuracy in open-domain question answering but at the cost of large space and memory requirements. |
| Approach: | They propose a simple unsupervised pipeline that includes principal component analysis (PCA), product quantization, and hybrid search to improve space efficiency. |
| Outcome: | The proposed pipeline achieves good accuracy–space trade-offs, for example, 48 compression with less than 3% drop in top-100 retrieval accuracy on average or 96 compression without drop in space requirements. |
An Open Multilingual System for Scoring Readability of Wikipedia (2024.acl-long)
Copied to clipboard
| Challenge: | Existing studies on the readability of Wikipedia have focused on English only and there are currently no systems supporting automatic readability assessment of the 300+ languages in Wikipedia. |
| Approach: | They propose a multilingual model to assess Wikipedia's readability using a dataset spanning 14 languages. |
| Outcome: | The proposed model outperforms existing models in a zero-shot scenario and is more accurate than previous benchmarks. |
HEAL: A Hypothesis-Based Preference-Aware Analysis Framework (2025.findings-emnlp)
Copied to clipboard
Yifu Huo, Chenglong Wang, Qiren Zhu, Shunjie Xing, Tong Xiao, Chunliang Zhang, Tongran Liu, JingBo Zhu
| Challenge: | Preference optimization methods like DPO are often evaluated on a single response, overlooking other outputs. |
| Approach: | They propose a Hypothesis-based PrEference-aware AnaLysis Framework that formulates preference alignment as a re-ranking process within hypothesis spaces. |
| Outcome: | The proposed evaluation paradigm re-ranks preference alignment as a reranking process within hypothesis spaces. |
From Zero to Hero: Human-In-The-Loop Entity Linking in Low Resource Domains (2020.acl-main)
Copied to clipboard
| Challenge: | Existing approaches to disambiguate entity mentions in a text depend on training data. |
| Approach: | They propose a domain-agnostic approach to annotate entities using a KB-based approach. |
| Outcome: | The proposed approach outperforms existing methods in a simulation on difficult texts. |
Generating Plausible Distractors for Multiple-Choice Questions via Student Choice Prediction (2025.acl-long)
Copied to clipboard
| Challenge: | Multiple-choice questions (MCQs) are critical for identifying misconceptions and gaps in knowledge and accurately assessing students' understanding. |
| Approach: | They propose to train a model to generate distractors that are more likely to be selected by students by a pairwise ranker and a distractor generator via Direct Preference Optimization. |
| Outcome: | The proposed model outperforms baseline models and performs comparable to humans in various metrics including pairwise rank accuracy and distractor plausibility. |
Ranking Unraveled: Recipes for LLM Rankings in Head-to-Head AI Combat (2025.acl-long)
Copied to clipboard
| Challenge: | Evaluating large language models (LLMs) is a complex task. Pairwise ranking has emerged as state-of-the-art method to evaluate human preferences. |
| Approach: | They propose to use pairwise ranking to evaluate human preferences . they propose to evaluate the robustness of ranking algorithms in LLMs . |
| Outcome: | The proposed methods are based on the principles of effective ranking and the robustness of several ranking algorithms in the context of LLMs. |
Beyond Averages: Learning with Annotator Disagreement in STS (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to capture and model disagreement in Semantic Textual Similarity (STS) ignore label dispersion and incentivize models to ignore uncertainty crucial for practical settings. |
| Approach: | They propose to capture and model disagreement in Semantic Textual Similarity (STS) a lightweight truncated Gaussian head and a cross-encoder are used to model disagreement . |
| Outcome: | The proposed approach improves accuracy and calibration of models to human judgments. |